30 research outputs found
Bi-directional Weakly Supervised Knowledge Distillation for Whole Slide Image Classification
Computer-aided pathology diagnosis based on the classification of Whole Slide
Image (WSI) plays an important role in clinical practice, and it is often
formulated as a weakly-supervised Multiple Instance Learning (MIL) problem.
Existing methods solve this problem from either a bag classification or an
instance classification perspective. In this paper, we propose an end-to-end
weakly supervised knowledge distillation framework (WENO) for WSI
classification, which integrates a bag classifier and an instance classifier in
a knowledge distillation framework to mutually improve the performance of both
classifiers. Specifically, an attention-based bag classifier is used as the
teacher network, which is trained with weak bag labels, and an instance
classifier is used as the student network, which is trained using the
normalized attention scores obtained from the teacher network as soft pseudo
labels for the instances in positive bags. An instance feature extractor is
shared between the teacher and the student to further enhance the knowledge
exchange between them. In addition, we propose a hard positive instance mining
strategy based on the output of the student network to force the teacher
network to keep mining hard positive instances. WENO is a plug-and-play
framework that can be easily applied to any existing attention-based bag
classification methods. Extensive experiments on five datasets demonstrate the
efficiency of WENO. Code is available at https://github.com/miccaiif/WENO.Comment: Accepted by NeurIPS 202
The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification
This paper introduces the novel concept of few-shot weakly supervised
learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC.
A solution is proposed based on prompt learning and the utilization of a large
language model, GPT-4. Since a WSI is too large and needs to be divided into
patches for processing, WSI classification is commonly approached as a Multiple
Instance Learning (MIL) problem. In this context, each WSI is considered a bag,
and the obtained patches are treated as instances. The objective of FSWC is to
classify both bags and instances with only a limited number of labeled bags.
Unlike conventional few-shot learning problems, FSWC poses additional
challenges due to its weak bag labels within the MIL framework. Drawing
inspiration from the recent achievements of vision-language models (V-L models)
in downstream few-shot classification tasks, we propose a two-level prompt
learning MIL framework tailored for pathology, incorporating language prior
knowledge. Specifically, we leverage CLIP to extract instance features for each
patch, and introduce a prompt-guided pooling strategy to aggregate these
instance features into a bag feature. Subsequently, we employ a small number of
labeled bags to facilitate few-shot prompt learning based on the bag features.
Our approach incorporates the utilization of GPT-4 in a question-and-answer
mode to obtain language prior knowledge at both the instance and bag levels,
which are then integrated into the instance and bag level language prompts.
Additionally, a learnable component of the language prompts is trained using
the available few-shot labeled data. We conduct extensive experiments on three
real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer,
demonstrating the notable performance of the proposed method in bag and
instance classification. All codes will be made publicly accessible
Systematic Assessment of Factual Knowledge in Large Language Models
Previous studies have relied on existing question-answering benchmarks to
evaluate the knowledge stored in large language models (LLMs). However, this
approach has limitations regarding factual knowledge coverage, as it mostly
focuses on generic domains which may overlap with the pretraining data. This
paper proposes a framework to systematically assess the factual knowledge of
LLMs by leveraging knowledge graphs (KGs). Our framework automatically
generates a set of questions and expected answers from the facts stored in a
given KG, and then evaluates the accuracy of LLMs in answering these questions.
We systematically evaluate the state-of-the-art LLMs with KGs in generic and
specific domains. The experiment shows that ChatGPT is consistently the top
performer across all domains. We also find that LLMs performance depends on the
instruction finetuning, domain and question complexity and is prone to
adversarial context.Comment: Accepted by EMNLP 2023 Finding
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Large language models (LLMs), such as ChatGPT and GPT4, are making new waves
in the field of natural language processing and artificial intelligence, due to
their emergent ability and generalizability. However, LLMs are black-box
models, which often fall short of capturing and accessing factual knowledge. In
contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are
structured knowledge models that explicitly store rich factual knowledge. KGs
can enhance LLMs by providing external knowledge for inference and
interpretability. Meanwhile, KGs are difficult to construct and evolving by
nature, which challenges the existing methods in KGs to generate new facts and
represent unseen knowledge. Therefore, it is complementary to unify LLMs and
KGs together and simultaneously leverage their advantages. In this article, we
present a forward-looking roadmap for the unification of LLMs and KGs. Our
roadmap consists of three general frameworks, namely, 1) KG-enhanced LLMs,
which incorporate KGs during the pre-training and inference phases of LLMs, or
for the purpose of enhancing understanding of the knowledge learned by LLMs; 2)
LLM-augmented KGs, that leverage LLMs for different KG tasks such as embedding,
completion, construction, graph-to-text generation, and question answering; and
3) Synergized LLMs + KGs, in which LLMs and KGs play equal roles and work in a
mutually beneficial way to enhance both LLMs and KGs for bidirectional
reasoning driven by both data and knowledge. We review and summarize existing
efforts within these three frameworks in our roadmap and pinpoint their future
research directions.Comment: 29 pages, 25 figure
ChatRule: Mining Logical Rules with Large Language Models for Knowledge Graph Reasoning
Logical rules are essential for uncovering the logical connections between
relations, which could improve the reasoning performance and provide
interpretable results on knowledge graphs (KGs). Although there have been many
efforts to mine meaningful logical rules over KGs, existing methods suffer from
the computationally intensive searches over the rule space and a lack of
scalability for large-scale KGs. Besides, they often ignore the semantics of
relations which is crucial for uncovering logical connections. Recently, large
language models (LLMs) have shown impressive performance in the field of
natural language processing and various applications, owing to their emergent
ability and generalizability. In this paper, we propose a novel framework,
ChatRule, unleashing the power of large language models for mining logical
rules over knowledge graphs. Specifically, the framework is initiated with an
LLM-based rule generator, leveraging both the semantic and structural
information of KGs to prompt LLMs to generate logical rules. To refine the
generated rules, a rule ranking module estimates the rule quality by
incorporating facts from existing KGs. Last, a rule validator harnesses the
reasoning ability of LLMs to validate the logical correctness of ranked rules
through chain-of-thought reasoning. ChatRule is evaluated on four large-scale
KGs, w.r.t. different rule quality metrics and downstream tasks, showing the
effectiveness and scalability of our method.Comment: 11 pages, 4 figure
A Survey on Temporal Knowledge Graph Completion: Taxonomy, Progress, and Prospects
Temporal characteristics are prominently evident in a substantial volume of
knowledge, which underscores the pivotal role of Temporal Knowledge Graphs
(TKGs) in both academia and industry. However, TKGs often suffer from
incompleteness for three main reasons: the continuous emergence of new
knowledge, the weakness of the algorithm for extracting structured information
from unstructured data, and the lack of information in the source dataset.
Thus, the task of Temporal Knowledge Graph Completion (TKGC) has attracted
increasing attention, aiming to predict missing items based on the available
information. In this paper, we provide a comprehensive review of TKGC methods
and their details. Specifically, this paper mainly consists of three
components, namely, 1)Background, which covers the preliminaries of TKGC
methods, loss functions required for training, as well as the dataset and
evaluation protocol; 2)Interpolation, that estimates and predicts the missing
elements or set of elements through the relevant available information. It
further categorizes related TKGC methods based on how to process temporal
information; 3)Extrapolation, which typically focuses on continuous TKGs and
predicts future events, and then classifies all extrapolation methods based on
the algorithms they utilize. We further pinpoint the challenges and discuss
future research directions of TKGC